Video display system

ABSTRACT

Disclosed herein are video display systems and methods. An exemplary video display system includes a computerized media processing unit configured to receive content including video, audio, graphics and internet content from a plurality of content source devices providing such content and configured to output a blended video signal receivable by a display device; and wherein the video signal output by the media processing unit comprises a plurality of arbitrarily shaped video containers all displayable on the display device simultaneously, each of the plurality of video containers capable of being positioned and displaying video content independent of the video content of the other video containers; wherein the video content displayed in one container may comprise a GUI for control of size and position of each of the other of the plurality of video containers in response to user commands.

PRIORITY CLAIM

This application is a continuation of international applicationPCT/US2014/053491, filed Aug. 29, 2014, and entitled “VIDEO DISPLAYSYSTEM”.

PCT/US2014/053491 claims the benefit of the following provisionalapplication: U.S. patent application 61/872,095, filed Aug. 30, 2013.

This application is a continuation-in-part of application Ser. No.14/871,535 filed on Sep. 30, 2015, which is a continuation ofapplication Ser. No. 13/454,610, filed Apr. 24, 2012, now issued as U.S.Pat. No. 9,210,361 on Dec. 8, 2015.

Each of the above-referenced applications is incorporated by referencein its entirety.

BACKGROUND

Field

The present invention generally relates to the field of video and audiocontent, control and display, including for television and other displaydevices.

Description of Related Art

Traditionally, television entertainment has been understood to be oneprogram being displayed on a television or display device like apersonal computer (PC) or mobile device. For example, a viewer may watcha movie on a large screen television that is being aired on a televisionnetwork. The viewer “tunes into” the network by changing a televisionstation or setting a cable box to the network channel. The program isoutput out of the set top box or other device and displayed on thetelevision with audio. If the viewer wants to see what is on anothernetwork, he changes the channel on the cable box (or other) and after amoment the television displays what is being transmitted on that channelor network.

Advertising and program structure is set up to break up the content of aprogram or movie to place other video messages, such as productadvertisement, local advertisement, channel advertisement, governmentemergency message, and the like. Typically, a viewer watches through theinterruption of the movie content and sees the advertisement or othermessage unless the user changes the channel while the program or movieis at break.

This advertising and single channel viewing set up often causes thebehavior of the viewer with the control of the television to start“channel changing” or “flipping” or “channel surfing”. Other causes ofhyper channel changing typically occur when a viewer wants to see otherlive events on other channels, a slow or boring section in the videocontent stream occurs, or for a host of other reasons. Ultimately“channel surfing” is viewer experience driven in their efforts tomaximize entertainment or information from the television. This behavioroften results in the viewer with the television control being the onlyperson to be able to watch or enjoy the television, often the family orthose in the household or group watching the television will disband toseparate display devices to “surf” to the video content of their choice,often exhibiting the same behavior.

Television viewing set up, especially for large screen flat paneltelevision viewing, is typically achieved by hooking up various videoinput devices, such as a DVD player, a cable box, a satellite box, avideo game console, an over the top video from the internet (OTT) box,and other inputs to the television. Then, by use of a select inputbutton on the television, the viewer selects the video input device thattakes control of the whole television picture. In some models thetelevision allows a “picture in picture” mode where video from twodevices can be displayed at one time with one small and one large. Thismode offers a bit of control, but not enough to satisfy most users.

The evolving changes in viewer sophistication, especially technicalsophistication, combined with significantly increased access to varioustypes and sources of video content has outpaced the ability of thecurrent television viewing and control model. To complicate the picture,often while the viewer is flipping and surfing, he/she is also on a PC,laptop, tablet, smart phone, or other mobile device to operateapplications, typically connected to the internet, for furtherexperience.

The entertainment industry including content creators, contentproviders, service providers, television manufacturers, advertisers andinternet companies are all currently limited in the ability to present acombined, easy to use and expandable television presentation that isoutside their scope of expertise. Therefore, what is needed is a systemthat considers all these combined interests and creates a new userexperience framework for television viewers.

Large scale, high definition television screens have become widelypopular in homes and in various other venues, including entertainmentand retail venues. Also, entertainment content sources haveproliferated, with a vast range of HD video, Internet and applicationcontent available. Some use has been made of “picture-in-picture”features, and some providers have channels that show a mix of the videocontent from some other channels, allowing a viewer to see what ishappening on another channel to allow timely navigation to the otherchannel; however, a need exists for improved interfaces that allow usersmore control over the on-screen experience, as well as the methods andsystems that enable such interfaces.

SUMMARY

The systems and methods contemplated herein allow for a new generationof entertainment possibility by decoupling the limitation of the videoinput sources that take control of the entire television screen forsingle video picture viewer experience. While “picture in picture” is afeature that allows viewing of two video contents on the sametelevision, it is too limited, because there is no ability to multiplexinput from more than one video input source (i.e. a cable box and videogame console, for example). Moreover, picture in picture does notprovide the flexibility required to expand the entertainment potential.The systems and methods herein enable an integration of Internet contentand video content. Further, this invention may allow a content creator,distributor, and/or owner to directly engage with an end viewer viainternet access over an internet video container at the same time thevideo content is broadcast. This engagement provides a newinfrastructure for new ideas to enhance entertainment, socialinteraction, advertisements, point of sale, and other benefits for theindustry.

In a first embodiment, a system is provided for multiplexing both aplurality of video content (live and/or deferred time) and multiplevideo access mediums or sources (such as cable television, satellite,video games, DVD players, closed circuit video, internet video such asclips, programming and movies, and the like.) The system achieves thisin a viewer friendly manner to enhance entertainment and otherentertainment possibilities. The output of a selected video content isseen in a container referred to herein as a video container. This videocontainer does not necessarily need to fill an entire screen of adisplay device. A plurality of video containers may be displayed on atelevision or other display device simultaneously, and may be positionedin various manners, including overlapping one another.

The system may include one or more inputs with access to one or morevideo programs. The video programs or video content may be live linearcontent such as broadcast television, cable/satellite stations, andothers, from various sources. The video content may also be video ondemand content such as movies that get delivered on command from aviewer. The video content could also be delivered over the internet orstreaming video. Any video source that can be displayed on a viewingscreen including video conferencing and security video could be inputsto the system. Additionally any entertainment media that could bedisplayed in or on an entertainment device, such as a PC, mobile deviceor television, could be a video content input; including digital radio,web page interaction, video game input and output and others.

The video container is a logical container that is displayed on thetelevision screen, and is an area where a particular video is played outand viewed. In many embodiments, the video container may besubstantially rectangular in shape to match the video content aspectratios, though the video container may be any shape. The video play outdoes not exceed the boundary of the video container. In someembodiments, the video container may be sized and resized, in nearreal-time to form different size bounding rectangles from small to aslarge as the screen of the display device. Preferably, the video picturemay scale to fit the bounding rectangle of the video container.Moreover, in some embodiments, the video container may be positioned andrepositioned in near real-time to put the frame of the rectangle inviewer selectable position on the television screen, includingoverlapping other video containers.

The video container may also have many properties that include but arenot limited to: video container type (global video container, sportsvideo container, movie video container, or web video container, etc.), acontent filter, a list of channels with location, point of sale data(like credit card info), permissions for viewer log-in, and others.These properties allow particular video containers to be programmed withproperties, and repeatedly accessed, which may enhance a userexperience.

In one embodiment, the properties of the video container may allow aviewer to confine and control content type in a particular videocontainer, which may enhance ease of use. For example, instead of“surfing” though an entire channel guide to find any sporting events, avideo container could be configured with a filter to display only livesporting events across all input services. In another example, a videocontainer may be configured to show only network news programming. Inthese embodiments, video container configurations may allow the viewerto quickly view all sports or news programs to find one of interest.

Video containers are independent from each other. Therefore, multiplevideo containers may be sized and positioned by a viewer for betterentertainment experience. For example, suppose a viewer wanted to watcha football game, two baseball games, and the news at the same time. Theviewer could launch the appropriate video containers for each of thefour desired programs, resize and position each in accordance to hisneeds.

In embodiments, a video display system includes a computerized mediaprocessing unit that is configured to receive video content from each ofa plurality of different source devices, such as a cable set top box, asatellite set top box, a gaming station, a streaming video set top box,a DVD player, a computing device such as a PC, laptop, tablet, or smartphone, or any device that provides output for a television, such as inHDMI format or an internet packet compressed streaming format, or thelike. The media processing unit is configured to aggregate and blenddifferent selected video content from the different source devices togenerate as output a blended video signal in HDMI format which isreceivable by a television. One of the source devices may be a networkcomputing device that is configured to run a web-based application thatprovides access to internet video content to the media processing unit,provides a graphical user interface for the video display system, andprovides control instructions to the media processing unit. Thegraphical user interface allows a user to select content from one ormore of the source devices to view on the television.

The video signal output by the media processing unit may comprise aplurality of video containers all displayable on the televisionsimultaneously with variable size and variable position, such as underthe control of a user, wherein each of the plurality of video containersis capable of simultaneously displaying respective video content fromone of the plurality of source devices, independent of the size,position, and video content of the other video containers. A videocontent displayable in a background video container on the televisionmay comprise the graphical user interface. The media processing unit isconfigured to control the size and position of each of the other videocontainers on the background container in response to controlinstructions of the web-based application. These control instructionsmay take the form of programmed instructions of the web-basedapplication (such as a predetermined or default layout of videocontainers) or user interactions with the graphical user interface tocontrol the size and/or position of each container.

The web-based application may run in a web browser of the networkcomputing device, or may be a native application running on the networkcomputing device, or may be an HTML, HTML5, java, javascript, Adobe orAdobe flash application. In embodiments, the network computing devicemay be a personal computer, a laptop computer, a tablet, a smart phone,or the like. The graphical user interface may be displayed on thetelevision or may be displayed on the network computing device runningthe web-based application, on another computing device, or on one of thesource devices providing content to the system. In embodiments, thegraphical user interface may be displayed on the television and mirroredin a computing device separate from the television for control of thesize and position of at least one of the plurality of video containersin response to user commands to the mirrored graphical user interface.The graphical user interface also controls the selection of content forat least one of the plurality of video containers displayable on thetelevision in response to user commands.

In embodiments, media processing unit may include a FPGA that isreconfigurable. For example, the FPGA can be reconfigured via softwarethat is downloaded over the internet, such as a web portal on theinternet, providing the ability to dynamically change a feature set ofthe hardware of the media processing unit such as to improve speed ofblending of the different video sources to produce unique blended videosignal outputs to the television.

A formatting module of the media processing unit is configured to resizethe content of each of the plurality of video source devices such thatit is displayable within a respective video container on the television.

A video container may be resized or repositioned based on activitydetected by the web-based application. A background video container maydisplay one or more widgets. A separate input device may be used toprovide control commands to the graphical user interface, such as aninput device that is associated with a respective one of the pluralityof content source devices. A plurality of input devices may be provided,each of the input devices under the control of a respective user forsimultaneously controlling a respective one of the plurality of videocontainers.

In embodiments, a video display system is providing that includes acomputerized media processing unit configured to receive video contentfrom each of a plurality of different source devices including at leastone of a cable set top box, a satellite set top box, a gaming station, astreaming video set top box, and a DVD player, and at least one networkcomputing device configured to run a web-based application that providesinternet video content to the media processing unit, provides agraphical user interface for the video display system, and providescontrol instructions to the media processing unit, the media processingunit configured to aggregate and blend different selected video contentand generate as output the blended video content as a video signal inHDMI format or another format which is receivable by a television orother display device. The video signal output by the media processingunit may comprise a plurality of video containers all displayable on thetelevision simultaneously with variable size and variable position,wherein each of the plurality of video containers is capable ofsimultaneously displaying respective video content selected from one ofthe plurality of source devices, independent of the size, position, andvideo content of the other video containers; and wherein the mediaprocessing unit is configured to control the size and position of eachof the video containers on the television in response to controlinstructions of the web-based application, which control instructionscomprise at least one of programmed instructions of the web-basedapplication and user interactions with the graphical user interface. Inembodiments, the video content displayable in a respective one of theplurality of video containers may comprise a portion of the videocontent from the respective source device up to the entirety of thevideo content from that source device.

In embodiments, a video display system may include a television and anetwork computing device for running a web-based application thatprovides a graphical user interface for the video display system andaccess to internet video content; and a computerized media processingunit configured to receive video content from each of a plurality ofdifferent source devices, wherein one of the plurality of source devicesis the network computing device for providing control instructions tothe media processing unit via the web-based application. The mediaprocessing unit is configured to aggregate and blend different videocontent from the different source devices and generate as output theblended video content as a single video signal in a format which isreceivable by the television. The video signal output by the mediaprocessing unit comprises a plurality of video containers alldisplayable on the television simultaneously with variable size andvariable position, wherein each of the plurality of video containers iscapable of simultaneously displaying respective video content from oneof the plurality of source devices, independent of the size, position,and video content of the other video containers; and wherein the mediaprocessing unit is configured to control the size and position of eachvideo container on the television in response to control instructions ofthe web-based application, which control instructions comprise at leastone of programmed instructions of the web-based application and userinteractions with the graphical user interface.

The system contemplated herein may also be configured to allow multiplepeople to privately listen to an audio of the video container ofinterest on the television while others watching the same television orother video display can listen to audio from different video containers.In one embodiment, the multiple audio outputs are transmitted viastreaming mechanism from the video containers system to mobile devicesover communications networks that are connected to the system, as wellas private headsets. In one embodiment of operation, different videocontainers may provide a particular, identifiable audio output channel.A user having a personal headset may connect the headset to an audiodevice in electronic communication with the system. The audio device maybe configured to change channels to provide access to the identifiableaudio output channels of each video container. In this way a user mayeasily access the audio channels of the various video containers. Audiodevices may include, but are not limited to computers, cellular phones,devices specifically configured for the present invention, televisions,video game consoles, and the like.

The system contemplated herein may further comprise a video camera, andmay be enabled to provide a video conferencing service allowing videocommunication between two or more users. In one embodiment, the videoconferencing service may allow a plurality of users to watch a livebroadcast and discuss the broadcast using the video conference system.The video conferencing system may be between two users, or may bebetween three or more users. In still another embodiment, a securitycamera feed may be displayed in a video container. In furtherembodiments utilizing social interaction aspects, a first user may senda video container from his display device to a display device of asecond user, allowing the second user to view what the first user isviewing on the sent video container. This feature may be tied intopackages such as the video conferencing service.

In one embodiment, properties of a screen may be manipulated by anetwork computing device. The network computing device is definedbroadly herein to include computing devices connected to a network, aswell as other input devices such as a remote control, PCs with web pageaccess to a web interface to the container system, body and/or handmotion devices, voice command devices and other control input devicesthat can be connected to video container system. In one embodiment, thesystem may be enabled to bring a video container into full screen modeupon command, and may return to a view showing multiple video containersupon command. In another embodiment, the video container system may beconfigured to have one large video container displayed, and a number ofsmaller video containers displayed on the same display device. In thisembodiment, the system may be configured to allow a video on a smallervideo container to be switched with a video on the large video containeron command.

Video containers may have many properties to control the bounds of thefunction the video containers. One such property is the display type. Avideo container may be a video display type that provides video play outfeatures and functions. Other display types are possible, one such typeis an internet type. The internet video container provides the samesizing and position capability but instead of video playout it mayperform web based application functions for example: HTML, Java Script,Java applets, etc. This allows a programmable application to be hostedon the internet and to be executed in the internet video container.

For example if a viewer wants to have a video container for all newschannels, the video container may have a list of news channels likeCNN®, Foxnews®, etc. and would not display or consider other stationslike the Disney® Channel because that is not news. The property of thevideo container may receive video programming across a variety ofsources, so for example the news channels may come in through a cablesource, a satellite dish source, and an internet source. All of thesesources that provide news programming may (or may not—depending on theproperties) be displayed in the video container. Another example is avideo container is configured to list the top 20 most popular videoclips on the internet for the current day. Each time launched the listis refreshed by video containers to find the top clip links for thatday.

In one embodiment, a video container and content may be streamed, bothaudio and video, to a mobile device, PC or other display device. In thisembodiment, the video content system may be configured as an integratedvideo access system providing video access to a plurality of devicessimultaneously.

In another embodiment, a single video container system may be operativewith a plurality of display devices, the system providing a plurality ofdifferent video containers to the different display devices. In afurther embodiment, a single control device may control the videocontainer content and layout on all of the plurality of display devices,and may allow transfer of one video container from one display device toanother. In one example, a single video container system may be employedat a bar having five large screen televisions. A single control devicemay control all of the video containers displayed on each of the fivetelevisions. Further, this control device may allow transfer of a videocontainer from a first television to a second television. In a furtherembodiment, the system may allow the five televisions to act as a singlelarge display and may allow movement of video containers as if thetelevisions were a single display.

The video container system structure also provides many opportunities toexpand advertising strategy and function. Control of video containerplayout and placement may allow spare television screen space foradvertisement. Further, interactive Web video containers may allowdirect viewer input for advertisement acknowledgement, purchase ofitems, point of sale, opinion input and more. Moreover, video or textadvertisements may be downloaded during video streaming and displayed invarious video containers at different times based on criteria andavailable display space and video container space. In one embodiment anadvertising module may be in communication with the system. Theadvertising module may provide a video content to the systemspecifically for advertisements. In further embodiments, a particularvideo container may be dedicated to the advertising module videocontent.

The video container system has an optional connection over internet to avideo container web site where the viewer may login and setup an accountto control features of the video container system. Features may includeability to choose if viewer analytics may be collected and sent to theweb account for use for dynamic advertising strategies and other uses.In one embodiment, the viewer analytics may be stored in a data storagemodule to store viewer data for access by a service provider. Thisconnection from a video container system to a computer in the networkalso provides the structure to enable social interaction with friends inreal-time, views of what friends are watching, what is popularprogramming from a group of social system users, and sharing of videocontainer settings including lists of popular video content channelsfrom distributed sources.

The present disclosure provides methods and systems that provide userswith improved control over what content is displayed on a screen, suchas a large format HD or 4k television screen, including providing theability to manipulate display location, sizing, transparency and otherfeatures related to a number of video sources of heterogeneous origin,such as television content, IP-based content, games, applications, andvarious other types of viewable content. The present disclosureprovides, among other things, a video display system for multiplexing aplurality of heterogeneous sources and a plurality of heterogeneouscontent into flexible, controllable, interactive video containers thatare displayed on televisions or other display devices for enhancedentertainment possibilities. In accordance with exemplary andnon-limiting embodiments, systems and methods for the simultaneousdisplay and interaction with multiple heterogeneous sources aredisclosed.

Also disclosed are methods of programming video containers by a usersuch that interactions with one container can result in alterations inrelated video containers; methods of programming video containers usinga user interface capable of adjusting the transparency of overlappingvideo containers that display wide format media from a single mediaprocessing unit; methods of programming video containers by providing auser interface having at least four display containers that can beplaced and sized via end user interaction. The user interface cancontrol the user experience. In embodiments, the user interface caninclude a background video container and other video containers canoverlay a user interface video container. The user interface may be oneof an HTML5 application running on a computing device such as a PC, suchas in a web browser or as a native PC application; and the userinterface can interact with another application or website to obtain oneor more of user preferences, widgets, and store analytics, and the userinterface can communicate with a formatting module of a media processingunit to control overlay and appearance of the video container outputdisplay.

Additionally, disclosed are methods of enabling an end user to adjustthe shape of video containers such the shape of the container isarbitrary, geometric, non-rectangular, or free-form in shape; methods ofenabling an end user to blend two or more heterogeneous sources, whereat least one source is video content and one is animation content, intoa single video container wherein a time sync may be established betweenthe video and the animation; methods of enabling an end user to blendtogether two or more video streams wherein the end user establishes thesynchronization between the two or more video streams. The content ofheterogeneous source types may be blended on the fly by relaying them toan output buffer on an FPGA (field programmable gate array) that is acomponent of a media processing unit.

Also disclosed herein are methods of overlaying blended content within abounded box region on a display screen that receives output created by adevice such as a media processing unit that handles multiple inputs,such as HDMI inputs or an HDMI output stream. A user interface can beprovided to allow an end user of a video display system with a displayscreen to overlay selected content within a bounded region ofpixels/video container on the display screen receiving output from amedia processing unit that handles multiple inputs, such as HDMI inputs,or an HDMI output stream.

In embodiments, a video display system may allow interaction withdisplayed video containers using a control interface or components of anative device that can provide a feed to one of the containers.

In embodiments, a video display system may comprise a computerized mediaprocessing unit configured to receive at least a plurality of contentincluding at least one of video, audio, graphics and internet contentfrom a plurality of heterogeneous content source types providing suchcontent and configured to output a video signal receivable by a displaydevice; and wherein the video signal output by the media processing unitcomprises a plurality of arbitrarily shaped video containers alldisplayable on the display device simultaneously, each of the pluralityof video containers capable of displaying video content independent ofthe video content of the other video containers; wherein the videocontent displayed may comprise, at a given instance, a portion of thevideo content from the source up to the entirety of the video contentfrom the source.

In embodiments, a user interface may be provided which has at least fourdisplay containers for display on a video display screen wherein thedifferent containers are capable of playing, at the same time,heterogeneous content types selected from video format content, HDTVformat content, packet-based content, video game content and audiocontent.

In embodiments, a method of inspecting a tag in an output stream createdfrom multiple input streams is disclosed in order to determine whatcontent was displayed on what part of the screen of a display device ata particular time, wherein the output stream can include an HDMI outputstream, and the input stream can include HDMI input streams. Thisinformation can be used for e-commerce purposes, or to provide_ananalytic report based on the use of the device.

In embodiments, methods of tracking can include filtering audio contentassociated with video content to be displayed wherein the filtersuppresses audio content in the human auditory range, summing thefiltered audio content with the audio content associated with the videocontainer of primary interest; and sending the combined audio to anaudio output device associated with a user.

In embodiments, a video stream can be provided that specifies a relatedstream of IP-based widgets that display themselves when the video plays.A widget can determine selection of a video stream from multiplepossible video streams when the widget is displayed on the video screen.Widget content can be shared with other users in a user-blended HDMIvideo and IP-TV environment. A widget can be recommended to users for auser-blended, large screen, HDTV environment based on similarity toother users. Video content can be recommended to users for auser-blended, large screen, HDTV environment based on similarity toother users. A personalized, contextual group of widgets that arepreferred by a particular user can be organized and displayed based onthe context of content displayed on the display screen from an HDMIoutput device that handles multiple HDMI input streams. A customizeduser interface can be provided which comprises one or more groupscomprising widgets, and internet browser windows, wherein a user is ableto flip between groups of widgets. A user interface can be providedwhich comprises one or more groups of widgets and internet browsers,wherein the selection of widgets within a group may be partially autopopulated based on the content of other video containers.

In embodiments, a system for processing multiple heterogeneous datasources having video content wherein the video content comprisesmeta-data embedded into the image content can include one or more ofinformation about the scene content of the video, links to relatedinternet sites, links to related social networking sites and channels,sales channels, and executable scripts. A transcoder module can extractthe meta-data from the image and act on the encoded meta-data accordingto a rule set.

In embodiments, a media processing unit or video display device mayinclude four or more input ports, one or more output ports, and an FPGA,wherein the input ports may be HDMI input ports, and the output portsmay be HDMI output ports. The FPGA may be a single FPGA, wherein theFPGA processes the individual inputs in parallel and writes theprocessed input to a memory buffer according to placement parametersspecified by a formatting module, wherein the contents of the memorybuffer are then output using one or more output ports. Hardware may beremotely upgraded using a communications network to upload new images tothe FPGA.

The FPGA may be programmable on the fly to enable at least one ofselection, blending and overlay logic for output of an HDMI stream tovideo containers of a display device while managing multiple HDMI inputstreams.

In embodiments, a custom/ASIC chip may take multiple HDMI inputs andprovide a single HDMI output stream, with parallel processing of inputstreams and a logic module for resizing and handling of overlayinformation among HDMI input streams.

In embodiments, a video display system handling multiple HDMI inputs andproviding a single HDMI output stream may include a synchronizationmeans for handling bit-by-bit competition on overlapping output regionsof multiple video containers displayed on a display device. A boundingbox of pixel locations may be provided for display of the HDMI outputstream of an FPGA to a display device. The system may convert content tofeed displays greater than 4k by combining multiple video streams andupconverting to a larger video output stream.

Video display systems described herein may be used for variousapplications. For example, a system for improved “TTY” communication maybe provided which includes two or more video containers wherein at leastone video container shows the TTY text and at least one video containershows a person.

In embodiments, a system for closed caption display may be providedwherein the closed captions are displayed in a separate, related videocontainer allowing the end user to scroll back through the closedcaptions, wherein an interface to the closed caption video container isthrough a controller or a secondary device such as a tablet or smartphone, and the user may control color and size of font, and position andsize of closed caption video container relative to related video.

In embodiments, a display screen may be provided having at least twovideo containers displayed wherein one video container displaysmarketing material comprising one or more of advertisements, videos,order forms, surveys, and games relating to the video content in arelated video container.

In embodiments, a method of providing and monitoring personalizedcontent as part of a video display system may include a website forstoring and serving personalized content including one or more of userpreferences, user information, information regarding user hardware,system and configuration information, storing and serving widgets fromcontent vendors based on user preferences and information; andcollecting and storing analytics. An user may perform one click shoppingat their television as part of the video container environment.

The present disclosure depicts a video display system that allows a newgeneration of entertainment possibilities by decoupling the limitationof various types of input sources, and related devices, such as set topboxes or video game consoles that each conventionally take control of anentire television screen or display for a single video viewerexperience. While “picture in picture” features and mix channels allowviewing of more than one video content stream on the same televisionscreen, these features are conventionally controlled by a single inputsource device and are very limited, among other things by the absence ofthe ability for the viewer to choose input from another device source,such as a cable box, video game console, satellite radio internet, DVD,RVU protocol or the like. In order to access other content, a user istypically required to select a different input source device, which thencontrols the entire display screen. Moreover, picture in picture and mixchannels are mainly limited to allowing a user to navigate to particularcontent in a timely way, not providing the flexibility of the methodsand systems disclosed herein.

Systems described herein decouple the display from the plurality ofheterogeneous inputs facilitating independent control of one or more ofscreen layout and video container characteristics, input devices and thelike. This control may use a multitude of control devices including oneor more of mobile devices such as smartphones, tablets and the like,PCs, remote controls, wireless mouse/keyboard devices and the like. Thesystem may receive input from one or more of these control devices andprovides universal remote control functionality enabling the user tocontrol the larger system comprising the display, the video displaysystem, the plurality of input devices, such as set top boxes, videogame stations, internet applications, internet set top boxes, videocameras, and the like,

In one embodiment, the system is able to accept and process multipleheterogeneous input sources simultaneously and merge them into acombined output stream in which the content of the different inputsources may be displayed in video containers having independent andarbitrary positions, sizes, aspect ratios, shapes, relative transparencylevels and the like. In a further embodiment the different input sourcesmay be linked such that the content of a video container may influenceone or more of the appearance, behavior or available controls of one ormore linked video containers. In a variation on the above embodiment,user interactions with a video container or its contents may result inchanges in one or more of appearance, behavior or options of linkedvideo containers.

These and other systems, methods, objects, features, and advantages ofthe present invention will be apparent to those skilled in the art fromthe following detailed description of the preferred embodiment and thedrawings. All documents mentioned herein are hereby incorporated intheir entirety by reference.

BRIEF DESCRIPTION OF THE FIGURES

The invention and the following detailed description of certainembodiments thereof may be understood by reference to the followingfigures:

FIG. 1 is a high level illustration of an exemplary video containersystem.

FIG. 2 provides an embodiment of the video container system configuredspecifically to have four video containers.

FIG. 3 provides a logical representation of how a video container may beresized and positioned on a television or other display device.

FIG. 4 provides a logical representation of how the video containersystem allows one or more mobile devices to connect to the system andselect an audio of a particular video container on the display device.

FIG. 5 provides an embodiment wherein the video container system ishosted by a remote server, which then provides video directly to a videodevice, such as a television or tablet computer.

FIG. 6 provides an embodiment demonstrating a variety of video andinternet video containers on a single display device—shown here as atelevision.

FIG. 7 provides another embodiment of the video container system havinga plurality of video containers arranged on a display device—shown inthis figure as a television.

FIG. 8 provides still another embodiment of the video container systemhaving a plurality of video containers arranged on a displaydevice—shown in this figure as a television.

FIG. 9 is an illustration of a tracking system for tracking a videocontainer of interest.

FIG. 10 is an illustration of playing audio associated with multiplevideo containers of interest.

FIG. 11 shows an example of a notch filter.

FIG. 12 illustrates an example of a display comprising video content anda group of widgets.

FIG. 13 shows an example of linked widgets and video containers.

FIG. 14 shows an example of using a single remote control for both thevideo display system and the heterogeneous input sources.

FIG. 15 shows an example of video display system with multiple inputports.

FIG. 16 shows details of the system hardware.

FIG. 17 illustrates the use of a user profile.

FIG. 18 shows an embodiment of the remote control widget.

DETAILED DESCRIPTION

System

FIG. 1 illustrates an exemplary embodiment of a video display system 46for presenting multiple views of content from heterogeneous inputsources 25 comprising one or more of audio, video, internet, computer,video game and the like that may be displayed on a television or displaydevice 38 or other display device and presented to one or more viewers.The multiple heterogeneous input sources 25 may be displayed in multipleindependent video containers on the television or display device 38 orother display device. In embodiments, the system supports four or moreindependent video containers on a video display screen wherein thedifferent video containers are capable of simultaneously playing ordisplaying content from independent, heterogeneous sources, such as fromseparate sources or source devices. A media processing unit 24 may beconfigured in whole or in part as a software program that runs on acomputer and the corresponding hardware may be implemented entirely insilicon or some combination thereof. The media processing unit 24 takesin content from the multiple heterogeneous input sources 25 and producesa video signal that a television or display device 38 may play forviewers in the various video containers.

Heterogeneous input sources 25 may comprise output from one or more of aset top box 23 receiving video content from a service provider, a DVD orvideo game console 29, a VHS 32, an internet video device 26 (connectedto IP network to Internet 28) such as a web cam, mobile device camera,personal computer camera, surveillance camera, video conference camera,video conference system or other camera, satellite feed, such as forsatellite television or radio, connected mobile device such as a tablet,smart phone or other device, local video devices such as a camera orbaby monitor, and the like. Content from heterogeneous input sources 25may be received via IP data streams received via Ethernet, coaxial cablesupporting Multimedia over Coax Alliance, MOCA, 802.11, 4G or othertransmission means and the like, wherein the IP data stream may becompliant with Universal Plug and Play, UPnP, Digital Living NetworkAlliance, DLNA, RVU and other guidelines, protocols, standards and thelike, over HDMI cable 31, output of a set top box 23, DVD or video gameconsole 29, VHS 32, internet video device 26, local video device, mobiledevices such as a tablet or smartphone or the like.

The media processing unit 24 controls the access to the heterogeneouscontent of heterogeneous input sources 25 by a variety of channel orcontent selection mechanisms including but not limited to HDMI channelchange commands over HDMI cable 31, Multicast IP leave/join overcommunications network 30, user interactions with a remote control, userinteractions with the native control interface of one of theheterogeneous input sources 25, and the like.

The media processing unit 24 takes commands from the viewer or viewersusing traditional remote controls, network computing devices 39 such asa mobile device like a tablet or smart phone, a computer, a gamecontroller, a wireless keyboard or the like. The network computingdevices 39 are in communication with the media processing unit 24 via acommunications network 22. Examples of such commands may be instructionsto resize, position, select content, link video containers, manipulatevideo container properties such as transparency and shape, manipulateweb content in video container, interact with local and web basedapplications and other control functions. Those commands determine howthe video container system will select content and present video andaudio to the viewer via the output to a television or display device 38.The network computing devices 39 may also provide video content or otherdisplayable content to the media processing unit.

FIG. 2 shows an embodiment of video containers configured specificallyto have four video containers 43, 42, 41,40 on a television or displaydevice 38. Video container 40 has a larger size and is positioned on theleft side of the television or display device 38. Three smaller videocontainers 41, 42 and 43 are positioned on a right side of the screenand each show different television channel video content. Videocontainer 40 may have its own properties configured by the user as towhat to watch, from which source, etc. In the embodiment shown, videocontainer 40 is showing a particular sporting event. Other sportingevents taking place at the same time are shown in video containers 41,42 and 43. In some embodiments, alternate camera views of the samesporting event may be shown in the smaller video containers 41, 42, 43,thereby providing multiple viewpoints of the sporting event. Alternatecamera views may further be employed with other video such as livetelevision content, enhanced movies, and programming configured topresent multiple camera views, among others.

FIG. 3 shows an embodiment of a logical representation of a videocontainer 44 being resized and positioned on a television or displaydevice 38. The network computing device 39 displays the video containerbounding box rectangle or video container representation 45. The networkcomputing device 39 such as a mobile or PC device may act as a userinterface and allow the viewer or user to resize the video containerrepresentation 45 shown on the network computing device 39, thentransmits a command over a communications network 22 to the mediaprocessing unit 24. The media processing unit 24 then changes the sizeand position of the video container 44 and matches the video containerrepresentation 45 on the television or display device 38. In oneembodiment, the network computing device 39 may have a touch screeninterface. In this embodiment, a user may utilize the touch screeninterface to move the video containers on the display device 38. Forexample, a user may “pinch” a representation of one video container onthe display device 38 to make it smaller. Similarly, a user may touchand “drag” a representation of a video container on the display device38 to move it across the display device 38.

FIG. 4 shows a logical representation of an embodiment of the systemallowing one or more mobile and/or PC devices to connect to the videocontainer system 46 and receive an audio output corresponding to aparticular video content. In this embodiment, a user may select aparticular video container view on the television or display device 38.The video containers system can then stream the audio for that videocontainer selected to the mobile or PC device that optionally may have aprivate audio headset. This may allow many people to listen to differentvideo container content without hearing other video container content.The present embodiment shows a first user listening to view 1 over acellular phone connected to a headset. A second user is listening toview 6 over a PC connected to a headset. A third user is playing a videogame and listening to the sound from the television speakers.

FIG. 5 shows an embodiment wherein the video container system is hostedby a remote server or “cloud”, which then provides video directly to atelevision or display device 38 such as tablet computer. An offsiteservice provider hub 50 may receive input video content 51 from one or aplurality of sources. This video may be received into a transcodermodule 53. A formatting module 52 may be in communication with thetranscoder module 53. The formatting module 52 instructs the transcodermodule 53 of what video streams to present, and what the proper formatand presentation of video containers is. The transcoder module 53 thenreceives the video content and instructions and outputs video formattedin various video containers as desired by a remote user. An over the top(OTT) streaming module 54 receives the formatted video container data,and outputs the same in a streaming data output to the internet, orother data connection. An end user may then receive the formatted videoover the internet on their television or display device 38. Control ofthe formatting and video content of the video containers may be achievedby transmitting a signal from a controller 55, over the internet orcommunications network, to the formatting module 52, which will instructthe transcoder module 53 to change its output, as discussed above. Thisembodiment may allow centralized control of the system, and may requireless hardware and setup for an end user.

FIG. 6 provides an embodiment demonstrating a variety of video andinternet video containers on a single display device—shown here as atelevision or display device 38. A large video container 70 ispositioned in an upper right corner of the television or display device38. The large video container 70 serves as the primary viewing area, andin this embodiment is showing a major network prime-time show. On a leftside of the television or display device 38 are three video containers71, 72, 73, which are smaller than the large video container 70. A topvideo container 71 displays a sporting event video. A central videocontainer 72 displays a news program, and a bottom video container 73may be used for “channel surfing” for example to find a video fordisplay in the large video container 70 during a commercial break.Further, at a bottom right portion of the television or display device38 is an internet video container 74. The internet video container 74 isshown accessing a webmail interface for checking email. This internetembodiment may allow a user to utilize a television as a single unit forinternet usage and television viewing, as opposed to a user havingmultiple devices such as a laptop and television.

It should be noted that the arrangement (including location and possibleoverlap with other containers), format and configuration of the variousvideo containers 70, 71, 72, and 73 may be varied in any number of ways,and is limited only to the size and technical limitations of the displaydevice.

FIG. 7 provides an embodiment of the video container system of aplurality of video containers arranged on a display device, shown inthis figure as a television or display device 38. The arrangement ofFIG. 7 provides video containers all relating to the same program, butfrom different video sources. A large central video container 80 ispositioned at a center right of the television or display device 38. Avideo content of this video container is a main camera view(s) of abroadcast television show. Three smaller video containers 81, 82 and 83are positioned at a left side of the television or display device 38. Atop video container 81 displays a video from the internet providing analternative viewpoint from what is displayed on the main program in thecentral video container 80. In one embodiment, the video container 81may display viewer video responses to a live broadcast in the centralvideo container 80. A middle video container 82 displays a videodisplaying a further alternative viewpoint from what is displayed on themain program in the central video container 80. A bottom video container83 displays a video displaying a further alternative viewpoint from whatis displayed on the main program in the central video container 80. Thecontent of the video containers 80, 81, 82, 83 may come from a varietyof sources such as a cable feed, internet, satellite dish, pre-recordedcontent, and the like. Further, the source of the content in the smallervideo container 81, 82, 83 may be different from the source of thecontent in the central video container 80.

FIG. 8 provides an embodiment of the video container system of aplurality of video containers arranged on a display device, shown inthis figure as a television or display device 38. The arrangement ofFIG. 8 provides video containers all relating to the same program, butfrom different video sources. A large central video container 90 ispositioned at a center right of the television or display device 38. Avideo content of this video container is a main camera view(s) of abroadcast television show. Three smaller video containers 91, 92 and 93are positioned at a left side of the television or display device 38. Atop video container 91 displays an interactive internet interface. Thisinterface may contain point of sale marketing, polling, voting,interactive gaming, and the like. A middle video container 92 displays avideo displaying an alternative viewpoint from what is displayed on themain program in the central video container 90. A bottom video container93 displays a video displaying a further alternative viewpoint from whatis displayed on the main program in the central video container 90. Thecontent of the video containers 90, 91, 92, 93 may come from a varietyof sources such as a cable feed, internet, satellite dish, may bepre-recorded, and the like. Further, the source of the content in thesmaller video container 91, 92, 93 may be different from the source ofthe content in the central video container 90.

Audio

One or more of the system's heterogeneous input sources 25 of FIG. 1 maycomprise audio content. The video display system 46 may be operable toautomatically transmit to audio output devices associated with thetelevision or display device 38 or display the audio associated with avideo container of primary interest. The identification of a videocontainer of interest may be derived based on one or more of thefollowing: selection by the user of a container of interest via a menu(on screen), switching among containers on screen via remote controluntil preferred audio is heard, data from input devices capable oftracking the gaze of one of more users to determine what container isbeing watched, or a rule set, such as one that indicates among variouscontainers default and/or customized (e.g., personalized) rules as towhat audio is preferred when multiple containers are present, and thelike. By way of example of a default rule set, when a containercontaining talk show content shares a screen with a container containingvideo game content, a rule could indicate that the talk show audiocontent will be heard unless the user takes action to select the videogame audio content, as one might conclude that it is unlikely a userwould watch a talk show without hearing the audio, while a user mightoften play a video game without sound. Similar default rules may bedeveloped for various pairs or larger groups of containers of content,and such rules might be customized to particular situations orpersonalized to preferences of particular users, such as by selection ofpreferences in a menu or by tracking user behavior to provide audioselection based on previous selections by the user in similarsituations. In embodiments, identification of the video container ofinterest may be based on length of gaze on a particular video container,a defined activation signal such as blinking twice, an activationmechanism set up by configuration, the size of the video container orthe like. As the apparent video container of interest changes the systemmay switch between audio content being played. The system may decode andblend the audio content of all the video containers and selectivelyreduce the sound levels of audio content not associated with the videocontainer. The blended audio may then be output to one or more oftelevision speakers, auxiliary speakers, mobile devices, blue toothheadphones and the like. For example, in embodiments the audio for thecontainer of interest could be sent to the main front speaker in asurround sound system, with audio for another container being playedthrough the back speakers, optionally at lower volume.

FIG. 9 shows a television or display device 38 with a tracking system202. The tracking system 202 may track the eye movements of one or moreviewers 204. Based on the tracked eye movements of the viewers the videodisplay system 46 may identify one or more video containers of primaryinterest. The system may be in a mode where a single video container ofprimary interest is identified based on the interest of one or moreviewers 204. In this mode a principal viewer may be identified, or theeye movements of more than one viewer evaluated according to a rule setto identify the video container of primary interest. If a single videocontainer of interest is identified, the associated audio content may beoutput to one or more of television speakers 208, auxiliary speakers210, mobile devices 212, blue tooth headphones 304 and the like.

Identification of the video container of interest may be based on lengthof gaze on a particular video container, a defined activation signalsuch as blinking twice, an activation mechanism set up by configuration,the size of the video container or the like. As the apparent videocontainer of interest changes the system may switch the audio contentbeing played to that associated with the current video container ofinterest. The system may decode and blend the audio content of all thevideo containers and selectively reduce the sound levels of audiocontent not associated with the video container. The blended audio maythen be output to one or more of television speakers 208, auxiliaryspeakers 210, mobile devices 212, headphones 304 and the like.

In some embodiments, the system may be capable of tracking an individualvideo container of primary interest for each of two or more users. Insome embodiments, the audio content associated with the video containerof primary interest for each user may be routed to an audio outputselected by the user such as one or more of television speakers 208,auxiliary speakers 210, mobile devices such as a tablet or smart phone,headphones 304 and the like. In an alternate embodiment, the audiocontent may be directionally output to the various users throughmechanisms such as directional audio speakers, wireless head sets andthe like.

FIG. 10 illustrates an embodiment where individual video containers ofinterest are identified for the one or more viewers 204 and respectiveaudio associated with each video container of interest is output suchthat it may be heard by the associated one or more viewers. The audiomay be output via an auxiliary speaker 210 creating directional audio302, headphones 304, a mobile device 308 or the like. The headphones 304may be connected with a wire, blue-tooth enabled or the like.

Some viewer monitoring services, such as offered by Nielson, listen foraudio signals embedded in content to determine what is being watched. Inone embodiment, the system may decode the audio associated with allvideo containers being displayed and pass all decoded audio through anotch filter. FIG. 11 shows an exemplary notch filter where frequenciesaudible to the human ear are suppressed and frequencies not audible byhuman ear are preserved. The filtered signals are then summed, togetherwith the full signal(s) associated with the one or more video containersof primary interest to the one or more viewers 204. The combined signalsmay then be re-encoded and sent to selected audio output devices asdiscussed above. An audio based tracking device may identify one or moreaudio signals, encoded at frequencies outside of those audible to thehuman ear, associated with the one or more heterogeneous source contentbeing displayed. The one or more users hear only the audio associatedwith the video container of primary interest.

Widgets

In one embodiment, a video container may comprise a widget where awidget is one of a website, an application running locally or remotely,a social media interface, a shopping application or the like. Two ormore widgets may be grouped in such a way that the user may interactwith the group of widgets as a whole. In some embodiments, there may bemultiple groups of widgets. The user may be able to flip between thedisplay of different groups of widgets. FIG. 12 shows an example of atelevision or display device 38 or other display with a group of widgets502 comprising one or more widgets 504. Group of widgets 502A iscurrently being displayed. Group of widgets 502B represents anadditional group of widgets that the viewer 204 may alternately selectto display. The system would enable the viewer 204 to flip between groupof widgets 502A and 502B. The widgets in a group may be personalizedbased on user identity, demographics, user preferences and the like. Thewidgets in a group may be partially preselected and grouped by the user.

The selection of widgets in a group may be partially auto-populatedbased on the content of other video containers. In some embodiments, thevideo stream may comprise metadata indicating widgets to be displayed.In other embodiments, widgets may be displayed based on a rule-set inthe video display system. An example of rule-based widgets may be thedisplay of groups of widgets based on the combination of genre beingdisplayed and user demographic and personal information. An example oflinking video content and widgets is shown in FIG. 12. In this example,the display comprises a video container having a baseball game and agroup or gang of widgets. The group or gang of widgets comprising one ormore of default and user specified widgets such as a weather app, astock market ticker and the like. Additionally, because there is abaseball game being shown in a video, the group of widgets may alsocomprise a link to team branded merchandise for the teams playing,statistics on the particular players on screen and the like.

In some embodiments, interaction with one or more of the widgets mayresult in a change in other video containers. The change may compriseone or more of the addition of video containers to the display, a changein the input being displayed in one or more on-screen video containers,the closure of one or more current video containers and the like. In anon-limiting example, selection of replay in a team widget may result inan additional video container showing the identified play being added tothe outgoing video stream. In another non-limiting example, a widget mayinclude a selection of video streams such as different screen angles ina sport game and selection of a camera angle might result in theaddition of a video container on the screen or a change in the cameraangle being displayed in an existing on-screen video container. In oneexample, a sports event may be shown in one video container with smallervideo containers in proximity showing alternate camera angles of thesame event. Selection of a video container containing one of thealternate camera angles may result in enlargement of that videocontainer and the reduction in size of the larger video containercomprising content related to the same sporting event, swapping ofcontent between video containers or the like.

A widget or group of widgets may be shared between users. A user mayrecommend various widgets or video content to others via socialnetworking channels, email, the Skreens website and the like. The usermay receive recommendations for widgets or video content based onsimilarities to other users where the similarities may includedemographics, user preferences, viewing habits and the like.

Content of video containers displayed on the screen may be changed usingone or more of: direct user initiation such as changing a channel orchanging a layout; user interaction with a widget resulting in changesto a separate yet linked video container; user interaction such aschanging the input or input channel of a video container which mayresult in a change in the group of widgets displayed, and an interruptfrom the internet which may initiate the opening of a new videocontainer or widget. An internet initiated, newly opened video containeror widget may contain an alert, a favorite user program, emergencybroadcast alerts and the like.

In some embodiments, widgets may be present in a cluster or gang ofwidgets but “hidden” from view a portion of the time and then “pop-up”asynchronously based on an internet signal, an external alert or event,content embedded in one of the heterogeneous sources, programmaticallyaccording to a rules set, or the like. In some embodiments, the widgetmay take action, such as appearing asynchronously, based on externalevents. The system may allocate resources to each video container andwidget regardless of whether the video container or widget is currentlyviewable in the output video stream. Upon resource allocation, a hiddenwidget may check for triggering events to determine whether to takeaction. In an illustrative example, a user may select an NBA widget.Among the user selectable options for the widget, may be the option topop up whenever one of the users preferred teams is ready for tip off.When this particular widget receives system resources it may check userpreferences and game status and take appropriate action such as displaya message that the game is about to start, open up a new video containerat a particular location on screen, change the input of a currentlyviewable video container to the game and the like. In other embodiments,a widget could poll for weather alerts, incoming calls, new email,urgent text messages, reminders, device alerts such as low battery on anoxygen tank and the like.

In one embodiment a widget may check search the text of the closedcaption meta-data which may be associated with video content. The textmay be analyzed for trigger words, phrases, repetition of key words andthe like. Closed caption text associated with the content of currentvideo containers may be searched as well as the closed caption textassociated with other channels of interest. A rule set may be compriseactions to be taken upon the identification of said trigger words andthe like. Actions may comprise the pop-up of a widget, opening a newvideo container, changing the content of an existing video container andthe like.

Some shopping widgets may have access to user profile information thatmay include shopping credentials. A widget specification may include anoption to simplify the purchase process and enable one click shoppingfor identified sites.

User Interface Capabilities and Hardware

The video display system of this invention is designed to provide a userwith extensive control over the viewing experience including one or moreof content, display layout, interactive properties and the like. A usermay be provided with tools to specify and control his or her viewingexperience with in-system control tools that may be displayed on thetelevision or display device 38 or display screen, such as in a videocontainer. The control tools may be manipulated using standard userinterface mechanisms such as a mouse, track-ball, joy-stick, motiontracking system, touch screen, or the like, or via a mobile device suchas a tablet or smart phone, PC, or the like. In another embodiment, theuser may specify viewing one or more of video container layout,parameters, user preferences and the like on a website or by using alocal or remote application. A user interface application may allow theuser to select from pre-set layouts or to explicitly and arbitrarilysize, position and shape each video container to be displayed. The shapeof a video container may be rectangular, circular, other geometricshape, free-form shape or the like. The user interface may enable thesizing, positioning and the like of four or more video containers. Theuser interface may allow the user to identify which video containershould be in front when two or more video containers overlay oneanother. Additionally, the user interface may allow the user to specifya degree of transparency associated with a plurality of video containerswhen two or more video containers overlay one another. FIG. 13 shows anexample of a television or display device 38 or other display device onwhich may be seen a plurality of video containers 602. The plurality ofvideo containers 602 comprises a circular video container 602A, anexample of one video container 602A overlaying and obscuring a portionof another video container. There is also an example of a partiallytransparent video container 602B.

In another embodiment, the user may be able to explicitly link thecontent of various video containers and individual or groups of widgets.The user may be able to specify inter-container relationships,interactions and the like in such a way that an interaction with onecontainer may result in a change in another video container, includingone or more of alternate content being shown, changes in size, position,shape, transparency, and visibility, alternate options available for thevideo container, and the like. An illustrative and non-limiting exampleof this would be where user selection of a genre in one video containerwould change the selection of channels available in another videocontainer.

In another embodiment, the user may be able to sync the input from twoor more heterogeneous input sources and then blend the output in asingle video container. Un-synced input might result in a mismatchbetween frames from the different sources. It may be desirable to syncthe inputs such that contiguous full frames from each source are blendedtogether rather that blending one source containing the last half of oneframe and the beginning of the next with another source that is showinga complete frame.

Another embodiment comprising synchronization of two or moreheterogeneous outputs may comprise blending two or more of video,animation content, game output, direct camera feeds and the like. Thismay allow a user to place themselves into action of the game, video andthe like.

User Interface Hardware

The system may support multiple input devices including mobile devicessuch as tablets, smart phones and the like, remote controls, PCs, gamecontrollers, mice, track-balls, joy-sticks, motion tracking systems,voice activation, speech recognition components, wireless keyboards, andthe like.

There may be multiple heterogeneous sources being displayed in videocontainers and visible as part of the combined output of the system. Theuser may wish to interact with the individual content of the one or morevideo containers. The user may also wish to interact with the system'sgraphical user interface to control layout, video container parametersand the like. This may be achieved by using one or more user inputdevices, such as network computing devices 39 of FIG. 1. In someembodiments, the user may interact with content of a video containerusing another input device, such as one associated with the source ofthe video container content, and may interact with the system using aninput device associated with the system. However, it may be confusingand time consuming for the user to switch between different devices.

Thus, in an embodiment of this system it may be possible for the user toutilize a single input device to control both the system parameters aswell as the various heterogeneous input sources. The system may be ableto toggle between interpreting received signals as inputs to the systemgraphical user interface (GUI) and as inputs to the input source of acurrently active video container. In an illustrative and non-limitingimplementation, the system may identify when a particular videocontainer has been selected. The system input device may or may not havean explicit option to select between system GUI control and control ofsource input. When the system receives commands intended for an inputsource, the signal is translated or remapped into a corresponding signalappropriate for the intended input source and resent using one or moreof a communications channel such as an HDMI command channel, aninfrastructure such as an IP network, IR laser, or the like, to theinput source that maps to the active video container.

In another embodiment, a third party remote control device or a controldevice for one of the heterogeneous input sources may be linked to thedisplay system and act as both an input device for the system GUI and asan input device for one or more of the heterogeneous input sources.Additionally, signals from such a third party remote control device maybe interpreted by the system as being for yet another input source. Inan illustrative example, the joystick for an xBox game system, one ofthe systems heterogeneous input sources, may also act as the controllerfor the system graphical user interface GUI, as well as a controller foran old VHS tape system, another of the systems potential heterogeneousinput sources.

In one embodiment, it may be possible to mirror the system's graphicaluser interface, GUI, on one or more secondary displays such as a tablet,a PC, a laptop, a mobile device, a smart phone, a second television, orthe like. This may provide the user with the ability to utilizeinterface capabilities that may be associated with the secondary displaysuch as a keyboard, touchscreen and the like. These additional interfacecapabilities may enhance the user experience and the user's ability toedit widgets, input user preferences and personal data, setup syncing ofa plurality of heterogeneous sources, adjust shape, size, andtransparency of video containers, enter complex statements for searchingor editing text fields and the like. The mirrored GUI may be implementedby serving the same video output to more than one television or displaydevice 38. Alternately, the same GUI and control may be achieved by anative application developed using a common API to that used for thesystem GUI.

In some embodiments, the system may support the simultaneous receipt ofGUI control information from one or more input devices. FIG. 14 shows anillustrative example of this embodiment with multiple input devicessimultaneously controlling a single television or display device 38 ordisplay device. FIG. 14 shows three viewers 204 watching a singletelevision or display device 38 or other display comprising a backgroundvideo container 602 and three additional video containers 602 displayingcontent independent of the other video containers. Each viewer may havea separate input device 702 being one or more of a mobile device 702Asuch as a table or smart phone, remote control 702B, PCs, gamecontrollers, mice 702C, track-ball, joy-stick, motion tracking systemand the like. With their individual input devices 702, each viewer 204may specify one or more of content, size, position, change in size orposition, transparency, and the like of one or more of the videocontainers 602. The viewers 204 may “battle” each other for the controlof all the video containers 602 or reach a compromise regarding controlover the individual windows/video containers 602. The individual viewers204 may each control the content, size, position, transparency, and thelike of one or more unique video containers. In some embodiments, eachviewer 204 may provide a unique user ID to the system, allowing thesystem to apply user preferences and associate user data to the videocontainer in their control including channels, color schemes, widgets,user purchase data and the like.

In another embodiment the multiple input devices may be used by theviewers to play games against one another, the computer, other users onthe internet, “televised” game shows and the like. In some embodiments,the game may be linked to game show that is televised, broadcast,digitally distributed and the like. A widget may be developed to play aparticular game show such as “Price is Right”, “Wheel of Fortune” or thelike. The live game may comprise metadata in addition to video and audiocontent. The widget may be designed to sync with a live game or aprevious game using show number or other identifying information. Thesynced widget may enable the viewers to play alongside of the televisedparticipants.

Hardware Configurations

In one embodiment of a video display system, shown in FIG. 15, there isa computerized media processing system 24 comprising four or more inputports 802, an FPGA 804, custom chip, programmable logic device or thelike and an output port 808. In addition, the embodiment may compriseadditional electronics hardware and software elements. Each input port802 may support receipt of data in a format such as HDMI, DVI, IP datastream comprising raw, encrypted, unencrypted, or other video dataschemes such as Universal Plug and Play, UPnP, digital living networkalliance, DLNA, RVU, and the like. The system may comprise input ports802 comprising support for one of more of the above input formats. Insome embodiments all the input ports 802 may be HDMI compatible. In someembodiments all the input ports 802 may be for the receipt of IP datastreams. IP data streams may be received via Ethernet, coaxial cablesupporting Multimedia over Coax Alliance, MOCA, 802.11, 4G or othertransmission means and the like. In yet other embodiments, there may bea mix of input port 802 types. The input ports 802 may be connected inparallel to a single FPGA 804 or other programmable logical devicewherein the programmable device is designed to process in parallel themultiple inputs in separately allocated processing areas 810 on thedevice. As the content from each input port 802 is processed inparallel, the subsequent output data streams are written to a commonmemory buffer 812. The common memory buffer 812 may also be located onthe same FPGA 804, logical device or the like. Processing the individualinputs in parallel on the FPGA 804 and storing the processed data on thesame FPGA 804 eliminates the need to move data across a computer bus andmay enhance processing speed. The contents of the common memory buffer812 are combined in conformance with parameters provided by theformatting module 814 and written to the output port 808.

In another embodiment, shown in FIG. 16 the computerized mediaprocessing system 24 may have twelve or more input ports 802 and an FPGA804. The input ports 802 may be connected in parallel to a single FPGA804 or other programmable logic device. In yet another embodiment, theremay be multiple input ports 802 configured in sets wherein the sets 902of input ports 802 are multiplexed 904 into a single FPGA 804 or otherprogrammable logic device. In this embodiment, at any given time, theFPGA 804 would be able to parallel process only the multiple input ports802 associated with a given set 902 of input ports 802. In addition tomultiple input ports 802, the computerized media processing system 24may have one or more output ports 808 comprising support for one or moreof HDMI, DVI, Internet Protocol, RVU, Universal Plug and Play, UPnP, andDigital Living Network Alliance, DLNA protocols and the like.

As the individual input ports 802 are processed, a portion or all of theoutput data stream for each input port 802 is written to a common memorybuffer 812. The specifics of what portion of each data stream is writtento the output buffer, the location at which the output may be written,the extent of overlay or blending of multiple heterogeneous data streamoutputs and the like are performed in conformance with parametersprovided by the formatting module 814 and a rule-set. The formattingmodule 814 may specify the display location for each input port 802including spatial location, size of display window, shape of displaywindow, overlap between video display containers including any overlay,transparency or blending of display windows. The FPGA 804 or processingmodule comprises circuitry capable of processing and combining themultiple inputs as specified by the parameters from the formattingmodule 814 and then sending the resultant, combined output to one ormore of the output ports 808.

The FPGA 804 or other processing module may comprise a logic module forresizing and handling of overlay information among input ports 802. Thesystem may handle multiple input ports 802 and comprise asynchronization module or process for handling or avoiding bit-by-bitcompetition on overlapping output regions of a display device. In oneembodiment the formatting module 814 may provide a bounding box of pixellocations for display to the output port 808 of an FPGA 804 or otherlogical device.

While the system for implementing this video display method has beendescribed in terms of certain hardware configurations these should notbe considered limiting. It should be understood that the functionalityof decoding and processing multiple input streams simultaneously anddisplaying the processed content in a set of video containers havingarbitrary parameters for size, shape and the like may be accomplished ina variety of ways. The decoding and processing may be done locally usingone of a variety of schemes such as a computer and software wherein thecomputer has sufficient processing power to process the multiple inputsin parallel, a combination of customized hardware and software, a customFPGA or a some combination thereof. Portions of decoding and processingmay take place in the cloud with the final combined output beingstreamed to the local display device. In one embodiment portions of thedecoding and processing may be distributed across various components ofthe larger system with some portions being done by a set top box, thetelevision, or the like.

In some embodiments, it may be possible to upgrade the firmware of theFPGA, or other programmable logic device. This may be done remotelyusing a communications network, locally using a DVD or other portablecomputer readable medium, and the like to upload a new image or data tothe FPGA.

In some embodiments, the upgrade of the FPGA may facilitate a rotatinghardware key system enabling user or system authentication prior toallowing licensed content to be displayed in one or more videocontainers. The use of a cloud or a website to maintain user profilesand information may enable the registration of the unit in combinationwith a user authentication widget. The video display system could thenauthenticate and enable the playback of digital rights management, DRM,protected media without the need to access individual websites forauthentication.

In some embodiments, the FPGA or other programmable logic device may beprogrammable in situ. This may allow updates to the rule sets for one ormore of blending, overlay logic and the like, security features, newtransport protocols for IP, HDMI or other transport systems for new waysto connect inputs to the system and the like.

In one embodiment, the video output of the system may be higher pictureresolutions, such as 4K. The system may generate this larger outputstream by combing multiple lower resolution video streams (such as 1080,720, and the like), upconverting to a single 4K or other higherresolution video stream and the like.

Video data is frequently transmitted in a compressed format such asMPEG2, MPEG4 encoded video and audio and the like. The compressionfacilitates transportation of large amounts of data across limitedbandwidth channels, for example video over cable to a home. However,once the compressed format data is received, processing is required toretrieve the uncompressed video and audio data. Additionally, processingmay be needed to decrypt additional encoding such as security, digitalrights management, DRM, and the like. Once the data has been thusprocessed it may be configured for output compliant with one or more ofthe following protocols; HDMI, DVI, Internet Protocol, RVU, UniversalPlug and Play, UPnP, and Digital Living Network Alliance, DLNA protocolsand the like. This data manipulation may require significant processingcapacity. Typical consumer video display equipment may only be equippedto process a single high resolution or possibly two low definition datastreams simultaneously.

Due to the limited capacity of typical existing consumer displayequipment it may be desirable to provide additional processing power tofacilitate the simultaneous processing of multiple compressed and orencrypted data streams. In one embodiment, this may be done byleveraging the HDMI control lines in conjunction with one or moreexternal processors such as HDMI stick PCs, PC, other Set Top Boxes andthe like. Using the HDMI control lines these external processors may beused as secondary devices to decode and decompress video and then outputusing the standard HDMI output lines.

In an illustrative and non-limiting example, the compressed video may bereceived by a primary set top box, STB, via cable, satellite, internetinfrastructure or the like. The format of this compressed video may beMPEG2 Transport Stream, a standard broadcast video standard, MPEG4, orthe like. The primary STB may have more than one tuner to select adesired program and receive the compressed data containing the audio andvideo of the desired program in an MPEG2 Transport Stream, MPEG4 datastream or the like. The compressed data may be sent by the STB over anout-of-band local internet connection, control channel in the HDMIprotocol, or other transport mechanism, to the video display system ofthis invention. The compressed data may then be sent to one or more ofthe secondary processing devices for decompression and decoding. Thedecompressed and decoded video and audio content may then be outputusing the HDMI interface.

In a variation on this embodiment, the use of external processingcapacity such as HDMI stick PCs, PCs, and other set top boxes may bereplaced by additional processing capacity on the FPGA. The FPGA orother programmable logic device, logic chip or the like may comprise oneor more decoders for decompressing MPEG2 Transport Stream, MPEG4 and thelike. The results of the decompression may then be transferred to one ofthe input port processing areas on the FPGA for integration into theoutput video stream.

In another variation of this embodiment, the FPGA, or other programmablelogic device, logic chip or the like may be incorporated into theprimary set top box, STB. In this embodiment, multiple tuners may beconnected by electronic bus (PCIe, parallel, etc.) to the FPGA. Thus,decompression is done in the set top box, STB, and the decompressedand/or decrypted video and audio content is then transmitted to thevideo display system.

In one embodiment, the system may transfer data between ports such asHDMI ports, IP Ethernet ports, and the like. In this manner, the systemmay send packet data between any program or device connected to thesystem such as being able to send data live from an internet widget toan xBox to be processed live and the like. The ability to transfer datadirectly between programs may reduce complexity including reducing theneed for extra wires, eliminating some remote controls, and the like.

In an illustrative and non-limiting example, statistics from a livebasketball game may be transferred via the video display system to anxBox attached to the system. The xBox may play a basketball video gameaccording to the statistics of the live game as received from the videodisplay system. This may enable a user to see, via an xBox basketballgame, a graphics rendition of real basketball game events where the gameis played according to the live statistics passed by a basketballwidget.

In an embodiment, one or more of the heterogeneous sources may comprisea “second screen” application enabling the system to displayapplications and content intended for display on a smart phone, table orother mobile device for display in a video container. In anotherembodiment, the system may mirror the display of a tablet, smart phone,or mobile device in a video container displayed as part of the combinedsystem output.

User Interfaces

In one example, one of the HDMI inputs to the media processing unit maycomprise a graphical user interface (GUI). This GUI may be displayed ina background video container over which additional video containers maybe displayed. This GUI may be one of an HTML5 application (with HTML5code) running in web browser, a native application running on the videodisplay device or the like. A user of the video display device or systemmay interact with this user interface through a variety of input devicesincluding one or more of mobile devices such as smartphones, tablets andthe like, PCs, remote controls, wireless mouse/keyboard devices and thelike. A user of the video display device or system may select a userprofile via the GUI causing the user interface to interact with anotherapplication, website or the like to access one of more of userpreferences, user associated widgets, personal user data includingusernames, passwords, and purchase information, user preferred layoutand channels and the like. Additionally, the identification of a uniqueuser may allow the system, either the video system or a remoteapplication, website or the like, to gather and store analytics on theparticular user, their viewing habits, nature of one or more of inputchannels being displayed, location, size, shape, transparency and thelike of each video container on the screen, which widgets are displayed,the number of widgets and the nature and extent of widget interactions,video game activity including game identity, length of play, and thelike, time and date information, which channel had corresponding soundbeing played, relationships between shows, widgets and the likedisplayed together, co-viewing of different material and the like. Theidentification of a unique user may enable the system to limit thedisplay of certain content, widget access, data collection and the likebased on age of user. The user interface may communicate with theformatting module to provide parameters relating to the layout andappearance of the output display.

The user interface may be accessed simultaneously by one or more inputmethods such as at the video display device, using a local applicationor a remote application running on a PC, mobile device or the like, oraccessing a website via a web browser on a device and a communicationnetwork, or the like. In embodiments, a user interface accessed anddisplayed in such a manner enables the creation of a profile of a uniqueuser including allowing the user to specify information about theirpersonal demographics such as name, gender, payment information such ascredit card, PayPal™ account information and the like. Additionally, theuser may specify preferred genres, favorite teams, news-stations, areasof interest, default screen layout including number, size, position anddefault content of video containers, widget preferences and the like. Insome embodiments, access to the different levels of customization may beavailable based on user enrollment in different business models. Inanother embodiment, the user interface may gather information about theparticular hardware, system and software configuration of the user'svideo display device.

FIG. 17 shows an embodiment, where user profile is selected, the videodisplay system may access a website or server via a communicationsnetwork 22 which would provide the particular video display system withone or more of the user's personal preferences 1002 including defaultdisplay layout, channels, widget selections and the like. The same oranother website may act as a widget server 1004 providing the latestwidgets to the video display system. Analytic data 1008 may also bestored in the cloud 1702.

In one embodiment, the HDMI input may comprise video content in whichmeta-data has been embedded into the video stream. The embeddedmeta-data may comprise information about the scene content of the videostream, links to related internet sites, links to related socialnetworking sites and channels, sales channels, executable scripts andthe like. In one example, the video content may be a televised game andthe embedded meta-data may include information about the team,information on the individual players, links to the team's website andthe like. In another example, the video content may comprise a news showand the embedded meta-data may include links to additional informationon the topic being discussed, information on the newscasters, secondarystories and the like. In a further example, the video content maycomprise a home-shopping channel and the embedded meta-data may containlinks to information about the individual items for sale, information ondesigners, information on related accessories and the like.

In one embodiment, the video display system extracts embedded meta-datafrom the video content of the plurality of input sources and acts on themeta-data according to a rule set. In one embodiment, the inclusion ofmeta-data may result in information scrolling across the bottom of thevideo container in which that video is displayed. In another embodiment,the meta-data and rules may result in a widget being displayed. Inanother embodiment, the meta-data and rule-set may result in a twitterfeed related to the video content being displayed on the larger screen.These examples are meant to be illustrative and not exhaustive.

Data Gathering

The system may track information about the viewing habits of users. Thedata gathered may be used to enable e-commerce, target advertising,collect analytics on device usage, and the like. In some businessmodels, background, content and service providers may base advertisingcharges based on how many people are watching a program as measured byNielson or a similar service provider. In one embodiment, meta-data,tags and the like embedded in the input content may be subsequentlyembedded in the output stream where it may be measured by a trackingdevice. In another variant, meta-data, tags or the like may be storedunder the direction of the media processing unit. Storage may be localto the video display unit or to a remote website or server via thecommunications network. This meta-data may be combined with informationincluding the nature of one or more of input channels being displayed,location, size, shape, transparency and the like of each video containeron the screen, which input channel had corresponding audio being played,which widgets are displayed, the number of widgets and the nature andextent of widget interactions, video game activity including gameidentity, length of play, and the like, time and date information, whichchannel had corresponding sound being played, relationships betweenshows, widgets and the like displayed together, time and dateinformation and the like.

In one embodiment, the system may monitor the quality of the networkquality, transport stream, corresponding content integrity, imagequality and the like. Network monitoring may comprise a widget thatmeasures bandwidth and quality of the network. The network widget may beinitiated by one or more of a user, a remote technician, an automaticscan system or the like. The widget may cause special tracer IP packetswith time stamps to be sent with the source content to measureturnaround back to the content delivery network, CDN, to measurethroughput, latency and the like. The transport stream sent over thenetwork may be one or more of MPEG2, MPEG4 encoded video and audio, andthe like. A monitoring widget may collect statistics, images and thelike and perform tests for speed, noise, quality and the like, andperform analysis of test results over time, time of day, and the like.These statistics may be sent to a central website or server over thecommunications network where collective analysis for errors in transportor content over regions of service may be performed.

Applications

In one embodiment, the system may be used for gaming. This embodimentcomprises internet widgets that accept bets over the internet or othercommunication networks and one or more video containers displaying livevideo content. Currently, users may use a PC, tablet, smart phone or thelike to visit one or more gaming web sites which enable users to seeoptions for placing bets, create an account, fund an account and placebets. These gaming web site may constantly update the odds, payout andclosed bets. These gaming web sites may have corresponding widgetsenabling bets to be seen and placed on the video display system of thisinvention. These widgets may enable asynchronous popups displayinginformation such as hot bets options, time running down for a game orrace to place a bet, results as they occur, and the like. System storeduser preferences and purchase data may allow for automatic login to thegaming widgets and corresponding gaming web sites.

In another variation, metadata about one or more sports events, races,game show, political race or other competitions and the like being shownin one or more video containers may be sent to a gaming widget. Thegaming widget may use information about the current sports event, race,or the like, information about upcoming events on a particular inputchannel, similar competitive events such as games, races and the like toidentify and show relevant bet options such as options for the currentcompetition, future games or races, multiple simulcast races, multiplegames shows with those multiple bet options shown, related competitiveevents and the like. The gaming widget with these options may bedisplayed as a popup if it is not already visible on the screen. Once abet has been placed, information regarding the bet may be displayed inproximity to a video container showing the event. Active and availablebets may be listed and selectable in the widget. Selection of specificbet or option may result in the relevant event, if available, beingdisplayed in a new or existing video container.

The system may display one or more live games, races, competitive eventsand the like, make grids of relevant content, and allow a user to resizecontent of interest. Alternatively, the system may auto size videocontainers based on the size or importance of bet relating to the one ormore live games, races, and competitions being displayed in the variousvideo containers. The system may visually identify the video containersdisplaying content where the viewer may have a winning or losing bet.Visual indicators may include a red ring around games viewer is losingand green around ones winning versus the bets made or similar visualindicators.

In one embodiment, the system may utilize one input source to take“snapshots” of each game across multiple channels. The system may tunein to each game of interest and take a picture or short video of eachand then display the “snapshots” in a gaming widget. The widget mayrotate through the series of “snapshots” enabling the user to watch andget a sense of the various games. The user may then select a “snapshot”to see the game of interest in a larger format resulting in the displayof the selected game in one or more of an existing video container, apop up second video container or the like.

The system may send user statistics, game statistics in combination withinformation on the content of visible video containers to a gamingwidget to facilitate mid game betting. In an illustrative example, avideo container displaying a basketball game that is in second quarterand an NBA widget with live statistics on the basketball game are bothvisible on the display. If one team is losing by 20 points the systemmay send information from the NBA widget such as the game score, inning,current outs and the like together with the game being watched to thegaming widget. The gaming widget may then pop up a mid-game bet giving+25 to the losing team. Viewer may then place bet. Additionally,statistics on individual players may be sent facilitating individualproposition bets such as how many steals a player will have, how manyrebounds, and the like.

In one embodiment local input and output devices such as cameras,joysticks, tactile stimulators and sensors and the like may be driven bywidgets and linked to content in other video containers such as videocontent, games and social interaction sites. These embodiments could beused for applications and content where tactile input and feedbackenhances the content such as learning musical instruments, physicaltherapy, adult content and the like.

In one embodiment, the system displays marketing or advertising materialbased on the content of displayed video containers. The system may openup a new video container to display the advertising material. The systemmay overlay a video container with a partially transparent videocontainer whose content comprises related advertising material. Theadvertising material may comprise one or more of advertisements, videos,order forms, surveys, games, promotional websites, linked socialwebsites, and the like.

In one embodiment, the system may comprise an improved TTY system forthe deaf. Currently one means of distance communication for the deafcomprises a TTY system where the user may place a “call” and thecommunication is accomplished via text being entered by the user andresponses seen on the users display. This embodiment comprises two ormore video containers wherein one would show the text similar to acurrent TTY system. However, an additional video container might show animage of the users. This would enhance communication since a significantamount of information is communicated non-verbally.

In another embodiment, the system may comprise an improved closedcaption system. Currently video content may include metadata with thetext of the audio content. In current systems opting to display theclosed caption content may result in the text of the audio beingdisplayed on the bottom of the screen. The text may obscure a portion ofthe video content and be difficult to read due to size, lack of contrastand the like. In this embodiment the selection to display the closedcaption information for a video source may result in a new videocontainer being displayed showing the closed caption text. The positionand size of this video container may be manipulated by the user for easeof viewing. It may be possible to scroll back through the content of theclosed caption text.

In another embodiment the system may comprise a home command controlcenter where the interface comprises a number of widgets and videocontainers. These widgets may enable one or more of asynchronous pop upsfor events such as door openings, sump pump status, motion detected,idle time on devices and the like, control options to manipulatedevices, display output from home video cameras while watching othervideo containers and the like. Control options available in such awidget may include turning a device on or off, locking windows and doorsremotely, turning on one or more alarms or sirens, sending a textmessage or audio or video phone call, teleconference or the like, eitherautomatically or with user input, to a person of interest such as thepolice, physician, a relative, security company or the like.

In one embodiment, an HDMI stick personal computer, personal computerprogram with wireless HDMI, or widget from web server may function as ahome control system monitoring one or more of security, home appliancestatus, and the like. In an illustrative example, a computer or webservice may receive data indicative of motion from an external homesecurity camera. This information may be sent to the video displaysystem and displayed as pop-up widget with a text message, video clip orthe like.

In one example the system may be programmed to power connected inputdevices on or off via HDMI or IP command based on one or more of currentdisplay status, idle time, user parameters and the like.

In one example the system may be programmed to cycle through a series ofcamera systems, web sites with cameras, baby monitors, door monitors andthe like. These camera views may be displayed in a background videocontainer while other video containers are displayed in the foreground.Audio may also be programmed to pop open a widget based on programmaticrules or external events. An illustrative example of audio may beswitching to audio from an infant's room on a periodic basis or when anoise is detected in the room.

In another embodiment, the user may scroll through images and videosfrom a tablet, smart phone, camera or the like connected to the system.The images may be shown in one video container while other video contentand widgets are displayed elsewhere on the screen. The system may beenabled to auto scroll through the images on the device without userintervention if desired.

In an embodiment, the system may comprise a system and method for one ormore of enhanced social commentary, social interaction and the likerelated to one or more of broadcast events, sports events, remotelyinteractive video games, online gaming and the like. In this embodiment,the system may obtain information about the content of a video containerthrough one or more of meta-data embedded in the video and extracted bythe system, information associated with channel selection and timeavailable in a database or online, or the like. Information about thecontent of displayed video containers, together with one or more of userpreferences, options, and analytics may cause a widget to open. Thewidget may display a social web site or channel related to the videocontent being displayed. The widget may be opened on the main displaydevice or on a secondary display device. The ability to interact withothers who are watching the same video content may create moreexcitement about the show through user interactions and sharedexperiences in cyberspace. Twitter hash tags may be automaticallyentered, extended and removed by other video container activity,including channel changes, new programs, history of surfing widgets,closed caption word or phrase or groups of words and the like.

In an illustrative example, users watching a reality TV show such asAmerican Idol, Iron Chef or the like, may chat together about therelative merits of the contestants, contestant appearance, opinionsregarding the contest and the judges and the like. The social site orchannel may collect these interactive comments, poll participating usersfor questions to be displayed on the show, filter people into smallertopic groups for augmentation of the show, and the like. It may bedesirable to utilize this information from users together with analyticsof viewer behavior and provide advertising specifically targeted tothese engaged viewers.

In another illustrative example, social commentary such as tweets andthe like could be incorporated into a video game as bubbles over a userscharacter, across the bottom of the screen or in a separate videocontainer. A game server could provide data regarding the relativeplacement for messages and the system could translate this informationto accommodate the actual size and location of the video containerdisplaying the game content.

In one embodiment the system may comprise Voice over IP, VOIP, on thetelevision. The system may allow voice calls to connect and play on thedisplay and through the associated audio output devices while watchingmultiple shows. The system may mute the sound of the TV in the audiothat is transmitted back to the caller but combine the sound from thecaller together with the previously selected audio output being sent tothe one or more audio devices. In one example, the user may be playingan online video game with remote friends and conversing with them viaVOIP simultaneously.

In another embodiment, the system may integrate the functionality of oneor more smart phones. This embodiment may include one or more videocontainers enabled to display a smart phone user interface. Thisinterface would enable a user of the video display system to engage insmart phone activities and applications such as texting, FaceTime™,Skype™ and the like via display of the smart phone user interface. Thesystem may also allow multiple video and voice calls simultaneously tomultiple video and widget activity on the screen.

In one embodiment, the system may accept input designed to be blended toproduce a stereo or 3D effect. This content may include meta-datadescribing the stereo effect to be achieved, which view, left or right,is represented and the like. The system may then cause the two or morecontent streams to be synced, blended and presented to the user. In somecases the output device may be a 3D television, stereo glasses of thelike. In some cases the system may represent the views in red and greenseparations for use with colored glasses.

In an embodiment, the system may act as a virtual living room or virtualbar, or the like. A widget running in full screen mode and logged into a3D virtual room may position a character or avatar on the screen torepresent one more of the user and others who may be logged in to acommon social site or the like. The users may manipulate the avatars insuch a way that they walk around the virtual rooms. A voice over IPconnection may allow users to communicate verbally with each other. Inanother embodiment, the system may blend live video over the 3D roomwhere the live video may be manipulated for one or more of size,proportion and angle to approximate the virtual environment. In oneimplementation position coordinates such as x,y, and angle may beprovided by the virtual environment widget. The system may then displaythe video at the location and in the orientation indicated in such a waythat the user may see real video blended into a virtual environment. Inan example, the system may be emulating a virtual sports bar. The systemwould insert the video of the game at the appropriate position with thevirtual sports bar. A group of friends participating in this virtualsports bar may be each be represented by an avatar and interact with oneanother. The virtual reality widget could provide parameters that wouldenable the system to keep the video content of the different systemssynced so that all users are seeing the content simultaneously.

In one embodiment the system may comprise a video conference system. Thevideo conference system may display a plurality of video containerswhose input comprises one or more of live broadcast video, video andaudio input from two or more sites, mirrored computer screens and thelike. In some embodiments the live broadcast video may synchronizedbetween the different sites in such a way that all participants areviewing the same content simultaneously.

In one embodiment the system may comprise a financial desk such as thatused by financial traders and the like. This embodiment would comprisemultiple video containers whose input comprises one or more of livebroadcast video such as financial news, current stock prices and one ormirrored computer screens and the like. The ability to display multiplelive video feeds and the computer desktop on the same screen mayfacilitate the ability of the user to process information without havingto switch from screen to screen for different information. Using thevideo display system of this invention allows the screen to display boththe broadcast video and the computer desktop without unduly slowing downthe computer.

In one application of this system the video display system may be placedin medical care environments such as hospitals, doctors offices, urgentcare centers, rehabilitation centers, nursing homes, home health careenvironments and the like. In addition to the heterogeneous inputsources described above, the input of the one or more video containersmay comprise one or more of output from various medical devicesincluding device status, physiological measurements such as bloodpressure, heart rate and the like, relevant instructional content fordevices, patient condition and the like. A video container may comprisean audio, or audio-video call with a health care provider where the callmay be initiated by one or the patient or other user or based on a ruleset, data received from external devices and the like. Device statusmessages may comprise informational messages such as low battery, lowfluid or oxygen messages and the like.

In one embodiment the system may comprise one or more remote controlwidgets. A remote control widget may provide functionality associatedwith a remote control associated with one of the input sources. In someembodiments the remote control widget would have the look and feel of aremote control associated with a specific input source. A remote controlwidget may comprise functionality for remote controls associated withmultiple input source devices. There may be distinct remote controlwidgets for each remote control associated with an input source. Uponselection of a video container, a remote control widget may “pop-up”displaying one or more of appropriate remote control functionality suchas channel controls, sound controls, menus and the like, remote controlappearance and the like. A remote control widget may be implementedusing HTML code, JavaScript, JSON, and the like to send control signalsto the intended source input. The controls may be sent using one or moreof the following methods: by the widget directly through the input portconnected to the input source device, via an IP command over thecommunications network or by sending a request to a cloud based serveror website which may initiate a web call to the intended input source.

FIG. 18 shows an embodiment of the remote control widget. The televisionor display device 38 or other display comprises three video containers602A 602B and 602C with content from different heterogeneous inputsources 25A 25B and 25C respectively. Currently video container 602C isthe primary video container and the remote control display widget 504Cis displayed. Some of the input ports may support 2-way communicationprotocols such as HDMI, IP and the like. The video display system 46 maysend control commands to these heterogeneous input sources 25 usingthese connections. In some instances the video display system 46 maysend the input controls to a 3rd party server or web site using thecommunications network 22.

In some instances the remote control widget 504 may be displayed on asecondary input devices such as a mobile device such as a tablet orsmart phone rather than on the television or display device 38 or otherdisplay.

In this disclosure, the term “video content” is used consistently withits use by those skilled in the entertainment technology to refer tovideo and optional audio programming regardless of transport, encryptionor other mechanisms that relate to how the video content is distributedor protected. For example, a movie is video content such as a newsbroadcast, internet video clip, video conferencing, or video from asecurity camera. Moreover, the terms “video” and “video content” aredefined broadly to additionally apply to internet browsing such as webpage viewing and other internet applications, email viewing, closedcircuit video, security camera video, and other displayable and/orinteractive content. The term “computer” is meant to encompass aworkstation, personal computer, personal digital assistant (PDA),wireless telephone, or any other suitable computing device. Terms suchas “component(s),” “device(s)” and the like are intended to refer tocomputer related entity, either hardware, a combination of hardware andsoftware, software or software execution. For example, a component maybe, but is not limited to being, a process running on a processor, aprocessor, an object, a reconfigurable hardware object, an executable, athread of execution, a program and a computer. By way of illustration,both an application running on a server and the server (or controlrelated devices) may be components. One or more components may residewithin a process and/or thread of execution and a component may belocalized on one computer and/or distributed between two or morecomputers or control devices.

The term “near real-time” refers to sensing, detecting, capturing and/orresponding to external events nearly simultaneously (e.g. withinmicroseconds or a few seconds) with their occurrence, or sufficientlyfast to meet the expectations of the viewer, such as change a channeland view the new channel video content.

Systems and methods embodying the present invention may be programmed inany suitable language and technology, such as Hypertext Markup Language(HTML), Active ServerPages (ASP) and Javascript. Alternative versionsmaybe developed using other programming languages including but notlimited to; C++, Visual Basic, Java, VHDL, other reconfigurable hardwarecomputer languages, VBScript, Jscript, BCMAscript, XML and/or CGI. Anysuitable database technology may be employed, such as: Microsoft Access,Oracle Databases and the like.

While only a few embodiments of the present invention have been shownand described, it will be obvious to those skilled in the art that manychanges and modifications may be made thereunto without departing fromthe spirit and scope of the present disclosure as described in thefollowing claims. All patent applications and patents, both foreign anddomestic, and all other publications referenced herein are incorporatedherein in their entireties to the full extent permitted by law.

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software, program codes,and/or instructions on a processor. The processor may be part of aserver, cloud server, client, network infrastructure, mobile computingplatform, stationary computing platform, or other computing platform. Aprocessor may be any kind of computational or processing device capableof executing program instructions, codes, binary instructions and thelike. The processor may be or include a signal processor, digitalprocessor, embedded processor, microprocessor or any variant such as aco-processor (math co-processor, graphic co-processor, communicationco-processor and the like) and the like that may directly or indirectlyfacilitate execution of program code or program instructions storedthereon. In addition, the processor may enable execution of multipleprograms, threads, and codes. The threads may be executed simultaneouslyto enhance the performance of the processor and to facilitatesimultaneous operations of the application. By way of implementation,methods, program codes, program instructions and the like describedherein may be implemented in one or more thread. The thread may spawnother threads that may have assigned priorities associated with them;the processor may execute these threads based on priority or any otherorder based on instructions provided in the program code. The processormay include memory that stores methods, codes, instructions and programsas described herein and elsewhere. The processor may access a storagemedium through an interface that may store methods, codes, andinstructions as described herein and elsewhere. The storage mediumassociated with the processor for storing methods, programs, codes,program instructions or other type of instructions capable of beingexecuted by the computing or processing device may include but may notbe limited to one or more of a CD-ROM, DVD, memory, hard disk, flashdrive, RAM, ROM, cache and the like.

A processor may include one or more cores that may enhance speed andperformance of a multiprocessor. In embodiments, the process may be adual core processor, quad core processors, other chip-levelmultiprocessor and the like that combine two or more independent cores(called a die).

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software on a server,cloud server, client, firewall, gateway, hub, router, or other suchcomputer and/or networking hardware. The software program may beassociated with a server that may include a file server, print server,domain server, internet server, intranet server and other variants suchas secondary server, host server, distributed server and the like. Theserver may include one or more of memories, processors, computerreadable media, storage media, ports (physical and virtual),communication devices, and interfaces capable of accessing otherservers, clients, machines, and devices through a wired or a wirelessmedium, and the like. The methods, programs or codes as described hereinand elsewhere may be executed by the server. In addition, other devicesrequired for execution of methods as described in this application maybe considered as a part of the infrastructure associated with theserver.

The server may provide an interface to other devices including, withoutlimitation, clients, other servers, printers, database servers, printservers, file servers, communication servers, distributed servers andthe like. Additionally, this coupling and/or connection may facilitateremote execution of program across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more location without deviating from the scope ofthe disclosure. In addition, any of the devices attached to the serverthrough an interface may include at least one storage medium capable ofstoring methods, programs, code and/or instructions. A centralrepository may provide program instructions to be executed on differentdevices. In this implementation, the remote repository may act as astorage medium for program code, instructions, and programs.

The software program may be associated with a client that may include afile client, print client, domain client, internet client, intranetclient and other variants such as secondary client, host client,distributed client and the like. The client may include one or more ofmemories, processors, computer readable media, storage media, ports(physical and virtual), communication devices, and interfaces capable ofaccessing other clients, servers, machines, and devices through a wiredor a wireless medium, and the like. The methods, programs or codes asdescribed herein and elsewhere may be executed by the client. Inaddition, other devices required for execution of methods as describedin this application may be considered as a part of the infrastructureassociated with the client.

The client may provide an interface to other devices including, withoutlimitation, servers, other clients, printers, database servers, printservers, file servers, communication servers, distributed servers andthe like. Additionally, this coupling and/or connection may facilitateremote execution of program across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more location without deviating from the scope ofthe disclosure. In addition, any of the devices attached to the clientthrough an interface may include at least one storage medium capable ofstoring methods, programs, applications, code and/or instructions. Acentral repository may provide program instructions to be executed ondifferent devices. In this implementation, the remote repository may actas a storage medium for program code, instructions, and programs.

The methods and systems described herein may be deployed in part or inwhole through network infrastructures. The network infrastructure mayinclude elements such as computing devices, servers, routers, hubs,firewalls, clients, personal computers, communication devices, routingdevices and other active and passive devices, modules and/or componentsas known in the art. The computing and/or non-computing device(s)associated with the network infrastructure may include, apart from othercomponents, a storage medium such as flash memory, buffer, stack, RAM,ROM and the like. The processes, methods, program codes, instructionsdescribed herein and elsewhere may be executed by one or more of thenetwork infrastructural elements.

The methods, program codes, and instructions described herein andelsewhere may be implemented on a cellular network having multiplecells. The cellular network may either be frequency division multipleaccess (FDMA) network or code division multiple access (CDMA) network.The cellular network may include mobile devices, cell sites, basestations, repeaters, antennas, towers, and the like. The cell networkmay be a GSM, GPRS, 3G, EVDO, mesh, or other networks types.

The methods, programs codes, and instructions described herein andelsewhere may be implemented on or through mobile devices. The mobiledevices may include navigation devices, cell phones, mobile phones,mobile personal digital assistants, laptops, palmtops, netbooks, pagers,electronic books readers, music players and the like. These devices mayinclude, apart from other components, a storage medium such as a flashmemory, buffer, RAM, ROM and one or more computing devices. Thecomputing devices associated with mobile devices may be enabled toexecute program codes, methods, and instructions stored thereon.Alternatively, the mobile devices may be configured to executeinstructions in collaboration with other devices. The mobile devices maycommunicate with base stations interfaced with servers and configured toexecute program codes. The mobile devices may communicate on a peer topeer network, mesh network, or other communications network. The programcode may be stored on the storage medium associated with the server andexecuted by a computing device embedded within the server. The basestation may include a computing device and a storage medium. The storagedevice may store program codes and instructions executed by thecomputing devices associated with the base station.

The computer software, program codes, and/or instructions may be storedand/or accessed on machine readable media that may include: computercomponents, devices, and recording media that retain digital data usedfor computing for some interval of time; semiconductor storage known asrandom access memory (RAM); mass storage typically for more permanentstorage, such as optical discs, forms of magnetic storage like harddisks, tapes, drums, cards and other types; processor registers, cachememory, volatile memory, non-volatile memory; optical storage such asCD, DVD; removable media such as flash memory (e.g. USB sticks or keys),floppy disks, magnetic tape, paper tape, punch cards, standalone RAMdisks, Zip drives, removable mass storage, off-line, and the like; othercomputer memory such as dynamic memory, static memory, read/writestorage, mutable storage, read only, random access, sequential access,location addressable, file addressable, content addressable, networkattached storage, storage area network, bar codes, magnetic ink, and thelike.

The methods and systems described herein may transform physical and/oror intangible items from one state to another. The methods and systemsdescribed herein may also transform data representing physical and/orintangible items from one state to another, such as from usage data to anormalized usage dataset.

The elements described and depicted herein, including in flow charts andblock diagrams throughout the figures, imply logical boundaries betweenthe elements. However, according to software or hardware engineeringpractices, the depicted elements and the functions thereof may beimplemented on machines through computer executable media having aprocessor capable of executing program instructions stored thereon as amonolithic software structure, as standalone software modules, or asmodules that employ external routines, code, services, and so forth, orany combination of these, and all such implementations may be within thescope of the present disclosure. Examples of such machines may include,but may not be limited to, personal digital assistants, laptops,personal computers, mobile phones, other handheld computing devices,medical equipment, wired or wireless communication devices, transducers,chips, calculators, satellites, tablet PCs, electronic books, gadgets,electronic devices, devices having artificial intelligence, computingdevices, networking equipments, servers, routers and the like.Furthermore, the elements depicted in the flow chart and block diagramsor any other logical component may be implemented on a machine capableof executing program instructions. Thus, while the foregoing drawingsand descriptions set forth functional aspects of the disclosed systems,no particular arrangement of software for implementing these functionalaspects should be inferred from these descriptions unless explicitlystated or otherwise clear from the context. Similarly, it will beappreciated that the various steps identified and described above may bevaried, and that the order of steps may be adapted to particularapplications of the techniques disclosed herein. All such variations andmodifications are intended to fall within the scope of this disclosure.As such, the depiction and/or description of an order for various stepsshould not be understood to require a particular order of execution forthose steps, unless required by a particular application, or explicitlystated or otherwise clear from the context.

The methods and/or processes described above, and steps thereof, may berealized in hardware, software or any combination of hardware andsoftware suitable for a particular application. The hardware may includea general purpose computer and/or dedicated computing device or specificcomputing device or particular aspect or component of a specificcomputing device. The processes may be realized in one or moremicroprocessors, microcontrollers, embedded microcontrollers,programmable digital signal processors or other programmable device,along with internal and/or external memory. The processes may also, orinstead, be embodied in an application specific integrated circuit, aprogrammable gate array, programmable array logic, or any other deviceor combination of devices that may be configured to process electronicsignals. It will further be appreciated that one or more of theprocesses may be realized as a computer executable code capable of beingexecuted on a machine readable medium.

The computer executable code may be created using a structuredprogramming language such as C, an object oriented programming languagesuch as C++, or any other high-level or low-level programming language(including assembly languages, hardware description languages, anddatabase programming languages and technologies) that may be stored,compiled or interpreted to run on one of the above devices, as well asheterogeneous combinations of processors, processor architectures, orcombinations of different hardware and software, or any other machinecapable of executing program instructions.

Thus, in one aspect, each method described above and combinationsthereof may be embodied in computer executable code that, when executingon one or more computing devices, performs the steps thereof. In anotheraspect, the methods may be embodied in systems that perform the stepsthereof, and may be distributed across devices in a number of ways, orall of the functionality may be integrated into a dedicated, standalonedevice or other hardware. In another aspect, the means for performingthe steps associated with the processes described above may include anyof the hardware and/or software described above. All such permutationsand combinations are intended to fall within the scope of the presentdisclosure.

All documents referenced herein are hereby incorporated by reference.

What is claimed is:
 1. A video display system comprising: a computerizedmedia processing unit configured to receive video content from each of aplurality of different source devices, the media processing unitconfigured to aggregate and blend different video content from thedifferent source devices to generate as output a blended video signal inan HDMI format which is receivable by a television; wherein one of theplurality of source devices is a network computing device configured torun a web-based application that provides access to internet videocontent to the media processing unit, provides a graphical userinterface for the video display system, and provides controlinstructions to the media processing unit; wherein the video signaloutput by the media processing unit comprises a plurality of videocontainers all displayable on the television simultaneously withvariable size and variable position, wherein each of the plurality ofvideo containers is capable of simultaneously displaying respectivevideo content from one of the plurality of source devices, independentof the size, position, and video content of the other video containers;wherein a video content displayable in a background video containercomprises the graphical user interface, and wherein the media processingunit is configured to control the size and position of each of the othervideo containers on the background container in response to controlinstructions of the web-based application, which control instructionscomprise at least one of programmed instructions of the web-basedapplication and user interactions with the graphical user interface. 2.The video display system of claim 1, wherein the web-based applicationruns in a web browser of the network computing device.
 3. The videodisplay system of claim 2, wherein the web-based application is one ofan HTML, HTML5, javascript, java, adobe, and adobe flash application. 4.The video display system of claim 1, wherein the web-based applicationis a native application running on the network computing device.
 5. Thevideo display system of claim 1, wherein the network computing device isone of a personal computer, a laptop computer, a tablet, and a smartphone.
 6. The video display system of claim 1, wherein the graphicaluser interface is mirrored in a computing device separate from thetelevision for control of the size and position of at least one of theplurality of video containers in response to user commands to themirrored graphical user interface.
 7. The video display system of claim1, wherein the media processing unit comprises a FPGA that isreconfigurable.
 8. The video display system of claim 1, wherein thetelevision is a 4K television.
 9. The video display system of claim 1,wherein the graphical user interface also controls the selection ofcontent for at least one of the plurality of video containersdisplayable on the television in response to user commands.
 10. Thevideo display system of claim 1, wherein the media processing unitcomprises: a transcoder module configured to receive content from eachof the plurality of source devices and configured to output a videosignal in the HDMI format; a formatting module in communication with thetranscoder module, the formatting module configured to instruct thetranscoder module as to which of the plurality of source device contentto present in which one of the plurality of video containers, and as tothe form of the presentation on the display device of each of theplurality of video containers in response to programmed instructions oruser commands; and a streaming module in communication with thetranscoder module and the communications network, the streaming moduleconfigured to receive the formatted video signal output from thetranscoder module, and to output the video signal as a streaming dataoutput to the television.
 11. The video display system of claim 10wherein the formatting module is configured to resize the content ofeach of the plurality of video source devices to display within arespective one of the plurality of video containers.
 12. The videodisplay system of claim 1, wherein a video container is resized orrepositioned based on activity detected by the web-based application.13. The video display system of claim 1, wherein the background videocontainer displays one or more widgets.
 14. The video display system ofclaim 1 further comprising a plurality of audio devices, each of theplurality of audio devices in communication with the media processingunit, each of the plurality of audio devices configurable to select andprovide an audio output of a video content from one of the plurality ofvideo containers, the audio device capable of changing the audio outputto correspond to any one of the plurality of video containers in nearreal-time.
 15. The video display system of claim 1, further including aseparate input device for providing control commands to the graphicaluser interface.
 16. The video display system of claim 15, wherein theinput device is one associated with a respective one of the plurality ofcontent source devices.
 17. The video display system of claim 15,wherein the input device is one of a third party control device and adevice associated with the network computing device.
 18. The videodisplay system of claim 1, further including a plurality of inputdevices, each of the input devices under the control of a respectiveuser for controlling a respective one of the plurality of videocontainers.
 19. A video display system comprising: a computerized mediaprocessing unit configured to receive video content from each of aplurality of different source devices including at least one of a cableset top box, a satellite set top box, a gaming station, a streamingvideo set top box, and a DVD player, and at least one network computingdevice configured to run a web-based application that provides internetvideo content to the media processing unit, provides a graphical userinterface for the video display system, and provides controlinstructions to the media processing unit, the media processing unitconfigured to aggregate and blend different selected video content andgenerate as output the blended video content as a video signal in aformat which is receivable by a television; wherein the video signaloutput by the media processing unit comprises a plurality of videocontainers all displayable on the television simultaneously withvariable size and variable position, wherein each of the plurality ofvideo containers is capable of simultaneously displaying respectivevideo content selected from one of the plurality of source devices,independent of the size, position, and video content of the other videocontainers; and wherein the media processing unit is configured tocontrol the size and position of each of the video containers on thetelevision in response to control instructions of the web-basedapplication, which control instructions comprise at least one ofprogrammed instructions of the web-based application and userinteractions with the graphical user interface.
 20. The video displaysystem of claim 19, wherein the video content displayable in arespective one of the plurality of video containers may comprise aportion of the video content from the respective source device up to theentirety of the video content from that source device.
 21. A videodisplay system comprising: a television; a network computing device forrunning a web-based application that provides a graphical user interfacefor the video display system and access to internet video content; and acomputerized media processing unit configured to receive video contentfrom each of a plurality of different source devices, wherein one of theplurality of source devices is the network computing device forproviding control instructions to the media processing unit via theweb-based application, the media processing unit configured to aggregateand blend different video content from the different source devices andgenerate as output the blended video content as a single video signal inan HDMI format which is receivable by the television; wherein the videosignal output by the media processing unit comprises a plurality ofvideo containers all displayable on the television simultaneously withvariable size and variable position, wherein each of the plurality ofvideo containers is capable of simultaneously displaying respectivevideo content from one of the plurality of source devices, independentof the size, position, and video content of the other video containers,and wherein the media processing unit is configured to control the sizeand position of each video container on the television in response tocontrol instructions of the web-based application, which controlinstructions comprise at least one of programmed instructions of theweb-based application and user interactions with the graphical userinterface.